Distributed Inference of Overlapping Communities

Overlapping community detection plays a key role in statistical network modeling. Despite the importance, popular models such as mixed membership stochastic blockmodels (MMSB) [2] are often not applicable to real world massive networks due to limited speed and memory of a single computing node. In this project, we develop distributed inference for models that can discover overlapping communities in real networks. Specifically, we address three key challenges in distributed network inference: 1) reduce O(N) pairwise parameters to O(N) by choosing constrained variational formulation described in [7]; 2) minimize communication cost between processes using a vertex-cut algorithm to partition the network; 3) synchronize global states between processes via a protocol designed for continuous aggregation and synchronization. Experimental results demonstrate our ability to tackle large scale network inference tasks.